Apparatus and method for user evaluation

ABSTRACT

A system for user-initiated analysis of changes in symptoms of a patient is described. The system includes a storage device for storing data captured of a patient performing an action in response to a stimulus presented to the patient. The stimulus presented to the patient includes one or more instructions provided to the patient in accordance with monitoring of medication administration of the patient. The action performed in response to the stimulus presented to the patient includes the action of the patient performing a requested step indicative of proper medication administration and is indicative of a change in one or more symptoms associated with a disease and the overall progression of the disease, the storage device further storing one or more item selected from the group of: demographic information of the patient, disease progression information of the patient, and one or more medication characteristics of the medication subject to the medication administration monitoring process. The system further includes a user input module for receiving input from a user indicative of a request for information, and a processor for determining, in response to the user input request for information, and in accordance with the captured data of the patient performing the action in response to the stimulus, a correlation between two or more of the following: one or more of the determined stored reactions, the demographic information of the patient, the disease progression information of the patient, and the one or more medication characteristics of the medication, the correlation being used to determine efficacy of the medication; and an output module for returning to the user information comprising the determined correlation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/721,456 filed Aug. 22, 2018 to Hanina et al.titled “Apparatus and Method for User Evaluation; U.S. ProvisionalPatent Application Ser. No. 62/648,815 filed Mar. 27, 2018 to Hanina etal. titled “Apparatus and Method for Computational Diagnosis of a User;and U.S. Provisional Patent Application Ser. No. 62/645,671 filed Mar.20, 2018 to Hanina et al. titled “Apparatus and Method for Interactingwith a User”, the entire contents of each of these applications beingincorporated by reference herein.

FIELD

This disclosure relates generally to the application of one or morenovel monitoring systems, including one or more of visual monitoring,audio monitoring, or other sensor data monitoring of one or morecharacteristics of a user, and more particularly, to the correlation ofresults of these unique measurements to quantify changes in the healthor other status of the user, and to further allow for prediction offuture response to the application of a drug, other therapy, or anychange in conditions. The disclosure further relates to a systemcapturing the collected and correlated information, analyzing thatinformation to determine further correlations, and allowing a user toprovide input querying the stored data in order to provide informationrelated to the applicability of a particular medication to a potentialpatient, or a type of medication to investigate in order to address aparticular patient or disease.

BACKGROUND

When evaluating a medical patient, doctors or other healthcare providersperform a generally manual process. This process includes reviewingcurrent characteristics of the patient against a predetermined set ofstandards to determine any deviation from those standards. So, thehealthcare provider may take the temperature or blood pressure of thepatient, and compare to standard, acceptable ranges for each suchmeasurement. By running through a sequence of these comparisons, abattery of tests may be performed to evaluate the health of the patient.Similarly, neurological or psychiatric tests may be applied that requestthe patient respond to specific questions or tasks, so that theresponses to these questions can be used to determine a disease state ofa patient, for example.

For more complex evaluations, more robust sets of tests may beadministered, including a series of tests that together may provideinsight as to the health of the patient. If a patient is being evaluatedfor a blood illness, for example, a sequence of blood tests may beemployed, while if the patient is being evaluated for a mental healthissue, such testing may comprise a sequence of questions that have beenvalidated to allow for confirmation of a diagnosis of a patient. Bothmethods of surveying the patient's current condition compare datacollected to an accepted range for what is considered “normal.”

SUMMARY

In U.S. patent application Ser. No. 12/620,686, filed Nov. 18, 2009,titled Method and Apparatus for Verification of MedicationAdministration Adherence, abandoned; U.S. patent application Ser. No.13/558,377, filed Jul. 26, 2012, titled Method and Apparatus orVerification of Medication Administration Adherence, now U.S. Pat. No.8,781,856; U.S. patent application Ser. No. 12/646,383, filed Dec. 23,2009, titled Method and Apparatus for Verification of Clinical TrialAdherence, abandoned; U.S. patent application Ser. No. 13/558,380, filedJul. 26, 2012, titled Method and Apparatus for Verification of ClinicalTrial Adherence, now U.S. Pat. No. 8,731,961; U.S. patent applicationSer. No. 12/646,603, filed Dec. 23, 2009, titled Method and Apparatusfor Management of Clinical Trials, Now U.S. Pat. No. 8,666,781; U.S.patent application Ser. No. 12/728,721, filed Mar. 22, 2010, titledApparatus and Method for Collection of Protocol Adherence Data, now U.S.Pat. No. 9,183,601; U.S. patent application Ser. No. 12/815,037, filedJun. 14, 2010, titled Apparatus and Method for Recognition of PatientActivities when Obtaining Protocol Adherence Data, now U.S. Pat. No.9,293,060; U.S. patent application Ser. No. 13/189,518, filed Jul. 24,2011, titled Method and Apparatus for Monitoring Medication Adherence,currently pending; U.S. patent application Ser. No. 13/235,387, filedSep. 18, 2011, titled Apparatus and Method for Recognition of PatientActivities, currently pending; U.S. patent application Ser. No.13/674,209, filed Nov. 12, 2012, titled Method and Apparatus forIdentification, now U.S. Pat. No. 9,256,776; and U.S. patent applicationSer. No. 13/674,459, filed Nov. 12, 2012, titled Method and Apparatusfor Recognition of Inhaler Actuation, currently pending; the contents ofthese applications being incorporated herein by reference, the presentdisclosure is directed to systems, methods and apparatuses that allowfor complete control and verification of adherence to a prescribedmedication protocol or machine or apparatus use in a clinical trial ordisease management setting, whether in a health care provider's care, orwhen self-administered in a homecare situation by a patient.

The application of the testing batteries, as discussed above, takessignificant healthcare provider time, is subject to variability inaccordance with subjectivity in grading responses across differenthealthcare providers and may also be difficult to administer outside ofa doctor's office or medical clinic. Furthermore, the amount andcomplexity of information that is used in making such determinations maynot be processed individually by a human being. It is this rich set ofhistorical information, collected concurrent information (for example,related to microexpressions that are not independently perceivable by ahuman) and other information collected and processed by the system thatallow for the system of the present disclosure to provide results andanalysis that are far more in depth, robust, consistent, and free ofsubjective bias when compared with those achievable by a human operator.Therefore, it would be beneficial to provide an improved process thatovercomes these drawbacks of the prior art. The present disclosure isdirection to, among other things, a system for collecting informationfrom a large number of patients, processing this information to allowfor a determination of correlations between collected data and changesin symptoms or other characteristics of the patients, to allow forobservation of changes in future patients in order to then correlatewith progression of disease. Furthermore, by additionally correlatingmedication adherence with those same changes in symptoms, it is possibleto then predict responses to changes in symptoms by future patients inresponse to administration of particular medications. Additionally, bydetermining elements of the medication and correlating these elementssimilarly with changes in symptoms, not only the complete medications,but also aspects of the medication can be used to predict futureresponses to other medications (potentially new medication underdevelopment, by way of example) by different demographic groups ofpatients. Finally, by selecting particular symptoms or aspects ofdisease to be cured, medication combinations can be determined andperhaps generated that have the most likelihood of success.

In addition, this disclosure proposes a method for establishing apersonalized medical baseline from which to compare shifts from anycollected data. Furthermore, the present disclosure covers techniquesfor monitoring changes in a number of physical and other characteristicsthat are not visible to a human reviewer over time, and thereforerequires the aid of an artificial intelligence system so that managingpatient symptom changes as a holistic group of characteristics that maybe correlated to changes in a disease being monitored.

Therefore, in accordance with the present disclosure, a novel system fortracking progression of symptoms of a patient is provided. Throughbaseline normalization, the use of both passive monitoring duringmedication adherence monitoring, and active monitoring duringpresentation of material to the patient, the systems and techniquesdisclosed herein are able to precisely monitor the patient and changesto symptoms or other identifying elements associated with the patient.These preferably lead to the ability to perform a differential analysisagainst healthy population and longitudinal self-comparison.

In some embodiments, the system therefore performs the following: 1)creating a patient-specific phenotype of the patient (for example, howsevere a condition experienced by the patient is in context to all theother patients who have gone through the system); 2) determining a rateof decline or progression of the symptoms/disease for the particularpatient; 3) generating a calibration point for a specific medicationbased on effectiveness to impact that specific indication in anobjective manner; and 4) creating a scoring system to show effectivenessand safety of a medication in comparison to predecessor drugs or othertreatment options that have gone through the system.

In accordance with another aspect of the present disclosure, datacollected across multiple patients or users may be compiled to allow forfurther analysis. In addition to determining a user baseline, andaverages of changes over time for patients in a particular population,the collected information may be used to determine potential outcomes ofother medications along similar dimensions. Such dimensions may includeability to reduce obesity, reduce suicidal thoughts, improve cognition,reduce pain, improve concentration, changes in tremor, or any othernumber of changes that may be determined. Thus, in accordance withanother aspect of the present disclosure, the collected information ispreferably analyzed along a plurality of dimensions so that expectedchanges along each of those dimensions may be predetermined. Eachpatient can also be characterized along one or more of these dimensions.Other aspects of the demographic information for each of the patientsmay provide the ability to then recognize characteristic changes alongeach of those dimensions categorized by demographic of the patient.Thus, by collecting information across all patients, categorizing thedata by demographic, and analyzing the categorized data, predictions offuture expected responses across these same dimensions can bedetermined.

In a still further aspect of the present disclosure, by alsocategorizing responses to the administration of medication along thesedimensions by different of the demographic groups and determiningaspects of medication responsible for changes to the characteristicsalong these same dimensions, it is then possible to determine potentialfuture expectations of medication responsiveness based upon patientdemographics, along these same dimensions. By searching a desiredcombination of results, the tool can be used as a medicationidentification tool, providing a guide for medications to be used tocombat certain of the symptoms matching the dimensions.

Still other objects and advantages of the invention will in part beobvious and will in part be apparent from the specification anddrawings.

In general, in some aspects, the subject matter of the presentdisclosure is directed to techniques comprising several steps and therelation of one or more of such steps with respect to each of theothers, and an apparatus embodying features of construction,combinations of elements and arrangement of parts that are adapted toaffect such steps, all as exemplified in the following detaileddisclosure, and the scope of the invention will be indicated in theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the invention, reference is made tothe following description and accompanying drawings, in which:

FIG. 1 depicts an exemplary data capture and transfer system;

FIG. 2 depicts more detailed elements of the data capture and transfersystem of FIG. 1;

FIG. 3 depicts a hierarchy depicting availability of commoditized sensordata;

FIG. 4 depicts a first portion of a sample interview;

FIG. 5 depicts a second portion of a sample automated interview;

FIG. 6 is a representation of an exemplary facial action coding system;

FIG. 7 is a graph representing the output employing the facial actioncoding system of FIG. 6, while a data subject is viewing an image of acar crash;

FIGS. 8A and 8B are graphs depicting an exemplary baseline shiftanalysis;

FIG. 9 depicts an exemplary analysis hierarchy;

FIG. 10 is a graph representing an exemplary longitudinal analysis ofcompound data sources;

FIG. 11 is a graph depicting an exemplary response where happinessexpression is diminished during speech;

FIG. 12 is a graph depicting an exemplary response where happinessexpression is relatively absent during an interview;

FIG. 13 is a graph depicting a filtered version of the graph of FIG. 12;

FIG. 14 is a graph depicting an exemplary relative expression between a7-variable expression and a 2-variable expression;

FIG. 15 is a graph depicting an exemplary prosodic analysis in responseto a visual stimulation test;

FIG. 16 is a flowchart diagram depicting an exemplary process forcollecting and analyzing data collected; and

FIG. 17 is a flowchart diagram depicting an exemplary process fordetermining one or more follow up information to be collected.

DETAILED DESCRIPTION

A novel system for tracking progression of symptoms of a patient isprovided in accordance with one or more embodiments of the presentdisclosure. The use of both passive monitoring during medicationadherence monitoring, and active monitoring during presentation ofmaterial to the patient may be used to determine an individual baselinefor a patient, and therefore the techniques disclosed herein enableprecise monitoring of the patient and changes to symptoms or otheridentifying elements associated with the patient. These preferably leadto the ability to perform a differential analysis against healthypopulation and longitudinal self-comparison, and to determineprogression of disease for a patient in relation to others in the samepatient population, and also against the patient's own particularcustomized profile.

In general, in some aspects, the subject matter of the presentdisclosure recognizes the desire to be able to apply the above describedsystem, and the audio/video information captured by the system, toperform unique healthcare evaluations. These evaluations are preferablybased upon visual and other captured data that allows for an ongoinganalysis of patient behavior, and the ability to monitor changes insymptoms of a patient employing visual and other observation, and toconfirm a particular medical diagnosis, or changes in that diagnosisover time. Monitoring of such changes may be visually and otherwiseconfirmed based upon monitoring of a patient performance of one or morestandard activities, such as while using the above medication monitoringsystem to monitor proper medication administration, or during anotheraction typically asked of a patient (“passive monitoring”). Theseadditional passive monitoring actions may include speaking with ahealthcare provider, walking in a normal manner, holding a mobiledevice, taking a picture with a mobile device, speaking to a friend on amobile device, typing a message on a mobile device, time in answering acall on a mobile device, responding to an instruction, and the like.These may include any normal action to be performed by a user, and themonitoring of one or more aspects of the user while performing theaction. By employing a visual solution, this passive monitoring may beperformed without the use of any additional hardware such as a wearablesensor, and therefore can be conducted in a far more unobtrusive manner.

Alternatively, the system may ask the patient to perform a particularset of actions in response to a displayed or otherwise presentedstimulus. Thus, the patient may be shown one or more images, presentedwith one or more sounds, asked to respond to one or more questions, etc.that are specifically provided in order to test or record a particularresponse to the presented material. As opposed to passive monitoring,such active monitoring, while more intrusive, is able to be moreparticularly tailored to gather desirable information from a user as thecollected information is not limited as in the passive monitoringsituation. These questions may be provided in a flexible manner,dictated by prior response to the same or other questions by thepatient, or may alternatively be based upon a standard set of questionstypically presented to a particular patient population. While thecontent may be presented in a flexible manner, the method of datacapture (audio, video, temporal, etc.) is preferably highly-specific andprovides the context necessary to extract meaningful signals. The systemmay also perform multi-modal tests, thus allowing for monitoring ofother action or characteristics of a patient while making the primarymeasurements using the audio/visual active or passive monitoringtechniques.

Furthermore, in either scenario, baselines for the main components(speech, visual, gestural, and temporal) measures may be established,and then subsequent questioning may be determined based upon baselineresults. For example, based upon the results received at baseline, aparticular set of questions may be asked of the patient going forward.This method of “customizing” the question sets allows for the ability tobest track changes in patient symptoms, while reducing the burden on thepatient as much as possible. Not only may the question set becustomized, but the precise mechanism for recognizing responses may alsobe customized based upon baseline or subsequent responses provided bythe patient. For example, if a particular patient is determined to havesymptoms or a disease that may result in an increased tremor in theirhands, more attention to the hands may be provided in order to betterdifferentiate between changes in the tremor. Similarly, if a patient isdetermined to have symptoms that may inhibit the ability to swallow amedication pill, extra attention may be provided to the throat of thepatient, looking at swallowing micro movements, to better confirm pillswallowing.

Such processing may be employed across any measured parameter, andallows for significant customization of the system to not only differentpatient populations, but also to individual patients while reducing thepotential processing burden on the system.

Visual —An advantage of the present system is that by capturing andanalyzing the frames of video of a person's face directly after theperson is subjected to an event, such as viewing a provocative image orbeing asked a personal question, the system can quantify a person'sunique myographic signature in response to that stimulus. For example,it has been discovered with the present systems that, in someembodiments, persons with negative symptoms of schizophrenia may elicitsmaller changes in facial expression when shown both positive andnegative imagery as compared with a healthy population. Additionally,when asked to describe the imagery and how it makes them feel,schizophrenics will offer less vocal response with distinct prosodicfeatures than from healthy patients. By basing the patient's visualmyographic signature on a comparison to their own, personally indexedvisual baseline (established from a composite array of neutralexpression imagery collected from the daily dosing application), thesystem can better extract variation from normal for that particularpatient. In one embodiment, when five images are viewed weekly over thecourse of three months, persons with negative symptoms fromschizophrenia will show a marked increase in both expression and prosodyif the medication is effective in treating this type of ailment, whichwill help to separate placebo from drug effect in a clinical trial. Ofcourse, other numbers of images or length of time may also be included.

Automated Negative Symptom Assessment—It has further been determinedwith the present systems that by recording audio and video of a personwhile the system asks them a set of questions, persons with negativesymptoms of schizophrenia will elicit smaller changes in facialexpression when they answer. Additionally, schizophrenics with negativesymptoms will offer less vocal response with distinct prosodic featuresthan from healthy patients. When a battery of questions is asked overthe course of three months, persons with negative symptoms fromschizophrenia will show a marked increase in both expression and prosodyif a medication they are being provided is properly working.

The systems encompassed by the present disclosure may be configured tomonitor video and audio in response to stimulus presented to the patienton a mobile or other electronic device, and further may monitor thepatient's response employing other sensors, such as accelerometers, orthe like associated with the mobile device. In some implementations,unique testing is performed in order to elicit the desired responses,employing visual recognition and analysis of facial expressions, audioanalysis of changes in vocal characteristics, and recognition of othersensor input. These tests are performed at predetermined or flexibleintervals, and therefore provide a unique picture of symptom progressionof the patient in a manner not previously available.

Stimuli may be provided to the patient through conduct of an automatedinterview, when real time analysis of responses provided by the patientwill influence the progress of the automated interview, and may also inpart dictate the measurements and analysis to be performed by the systemon the patient. Through unique branching logic that relies upon resultsfrom the real time analysis of patient responses, rather than simply aprovided answer, more efficient and accurate monitoring and analysis maybe provided. Branching not only considers the answers provided, butalso, for example, the amount of time to respond, any delays in speech,the inflection of the voice of the user, measurements of levels ofexcitement or lack thereof, or any other number of non-traditionalbranching logic indicators, thus allowing for branching in response toperhaps a holistic view of the user, rather than simply their answer tothe prior question. This flexibility allows for the system to continuesuch an interview in a more conversational manner, keeping the patientmore comfortable while maximizing the amount of relevant informationextracted from the interview systems. Additionally, such an automatedsystem removes bias that may be introduced by a human interviewer, orbecause of inter-interviewer and intra-interviewer differences inpresentation or evaluation criteria. Accordingly, the presentlydisclosed unique systems provide objective and repeatable benefits. Theresult is a customized interactive experience for the user withobjective and quantifiable measurements. In one or more embodiments, thesystems of the present disclosure may employ a data-driven machinelearning based approach for modeling (feature extraction, selection,decision making, etc.) in order to analyze collected data, includingvisual data. For example, techniques may be used to model healthypopulation, and extract and select signature differential features inaccordance with different artificial intelligence models.

In some embodiments, monitored characteristics also are used to assistin confirming medication administration. Thus, combined with directmonitoring of medication adherence, the further monitoring of physicaland other characteristic that are expected to change in a known mannerupon administration allows for an indirect, supportive confirmation ofproper medication administration.

As is therefore shown in FIG. 16, an exemplary process according to thepresent disclosure is shown. One or more of the steps of the process maybe performed by the data capture and transfer system described indetailed below with reference to FIG. 1 and FIG. 2. The processincludes, as step 1610 first obtaining data related to one or morephysical or other attributes of a user, and may include audio and/orvideo data, vital signs, weight, or other physical attributes, responsesto one or more presented inquiries or the like. This data may becollected, e.g., through microphones and cameras associated with amobile device of a user. Upon collection of this information, processingthen passes to step 1620 where a baseline for a particular user isdetermined in accordance with the collected information. Once thisbaseline for one or more of the collected data types is determined,processing passes to step 1630 where, based upon the baselinedetermination, one or more types of follow up information are determinedto be collected. The follow up information may include, e.g., one ormore passive or active presentation and collection of data, as will bedescribed below, and may include collection of such data in accordancewith other activities to be performed (such as medicationadministration, “passive”) or in accordance with specific taskspresented to the user, preferably by display and speaker of the mobiledevice of the user (“active”). In some implementations, branching logicis employed in order to determine the follow up activities andinformation to be collected. Thus, baseline information may be employedto determine a possible disease or ailment of the user, and then moredetailed information may preferably be determined useful to furtherevaluation any such condition. At step 1640 this additional follow upinformation is collected, and at step 1650 this additionally collectedfollow up information is employed to update the base line collectedinformation (returning to step 1620), and may further be used todetermine if any still additional branching logic is appropriate toidentify even further desirable follow up information should becollected (returning to step 1630). Finally, at step 1660 a diagnosis orprogression of disease is determined based upon changes in one or moreof the collected information. Any such updated diagnosis may similarlybe used to modify baseline at step 1620, or further information to becollected in accordance with branching logic at step 1630. After suchdiagnosis is determined to be complete, processing may end.

Referring next to FIG. 17, an example process for determining one ormore follow up information to be collected, as described in step 1630 ofFIG. 16, is depicted. One or more of the steps of the process may beperformed by the data capture and transfer system described in detailedbelow with reference to FIG. 1 and FIG. 2. As is shown in FIG. 17,processing begins at step 1710 where baseline information is analyzed todetermine whether portions of the baseline information are indicative ofa particular disease of interest, or even whether a particular symptomis appropriate to be further monitored. At step 1730 branching logic ispreferably employed in order to determine whether follow up informationis to be collected. Then at step 1740 it is queried whether data isbeing collected in an active or passive manner. In some implementations,this can be important given that data collection techniques may dictatechanges to the processing that is employed. By way of example, if apassive data collection technique is to be employed, processing passesto step 1750 where the updated data extraction technique includeschanges in the analysis techniques employed on the passive data that iscollected. Because the data is collected passively, the data collectionprocess and stimuli cannot be altered, and updated data extraction isthe only possibly-employed process. Thus, while alternative dataanalysis can certainly be employed, the monitoring process cannot bealtered (i.e. by presenting a different image or instruction to a user)because of the passive nature of the data collection. It may bepossible, however, to add means of data capture in response to passivelycollected information. For example, if video capture is employed, and itis observed that the mouth of a user may be moving, audio mayselectively also be employed. Thus, while branching logic may be moreactively employed in active situations, there is possibility forbranching of the logic in passive situation as well, but not to theextent to adjusting stimuli presented to the user. Once determined,processing passes to step 1760, representative of step 1650 of FIG. 16,where baseline data is updated in accordance with this updated analysis.

If, on the other hand, it is determined at step 1740 that datacollection is active, processing passes to step 1770 and one or moreupdated data presentation and collection techniques are defined. Thus,new scales or other presentations of data and collection of responsesare defined, in a manner as noted below. Thus, if a user is determinedat baseline to have a particular ailment, a particular test may beadministered to the user to define more precisely the user's diseasestate. Branching logic may define multiple sequential changes to thepresented and collected data as disease or symptoms progress. By way offurther example, if a user is determined to have a mental illness, suchas negative symptoms schizophrenia based upon the baseline analysis, itmay be determined to administer additional, more in depth instruments tothe user in order to define more precisely the disease state, andprogression of the disease, of the user. Once a predefined additionalprogression has been observed, further presentation of material may beprovided that is determined to be better applicable to later stages ofsuch a disease. Once implemented, processing passes to step 1790, alsorepresentative of step 1650 of FIG. 16, where baseline data ispreferably updated in accordance with this updated data presentation andcollection.

Information Capture System

Referring next to FIG. 1, a data capture and transfer system constructedaccording to the present disclosure is shown. In FIG. 1, a remoteinformation capture apparatus 100 is first shown. Such informationcapture apparatus 100 is adapted to allow for the capture and processingof information in order to implement the system and method in accordancewith the present disclosure, such as capturing one or more images of apatient administering medication, responding to presentation of one ormore images or other stimuli to the patient, or conducting an adaptive,simulated interview with the patient. Such information capture apparatus100 is preferably placed in communication with a remote data andcomputing location 300 via a communication system 200, preferably theInternet or other communication system. Via communication system 200,information captured by apparatus 100 may be transmitted to remote dataand computing location 300, and analysis information or otherinstructions may be provided from remote data and computing location 300to apparatus 100.

Remote data and computing location 300 may further process informationreceived from information capture apparatus 100. Such processedinformation may preferably be provided to a remote information displaydevice 400 via communication system 200. Remote information displaydevice 400 is further adapted to receive input from one or more users toprovide the received input back to remote data and computing location300, via communication network 200, in order to direct the processing ofreceived information by remote data and computing location 300. Theprocessing by remote data and computing location 300 may further beconducted in accordance with information received from informationcapture apparatus 100, information pre-stored to remote data andcomputing location 300, and other information that may be provided viaremote information and display device 400.

One or more patients of a plurality of patients may employ an individualinformation capture apparatus 100 in order to provide informationspecific to that patient to remote data and computing location 300.Therefore, in addition to capturing data related to the activitiesperformed by the patient in response to one or more prompts provided tothe user patient information capture apparatus 100, each correspondinginformation capture apparatus 100 may capture one or more passiveactivities performed by the patient while the patient engages theinformation capture apparatus, as will be described. Similarly, one ormore remote information and display devices of a plurality of remoteinformation and display devices 400 may be employed by a correspondingone or more users, each requesting different processing by remote dataand computing location 300.

It is further contemplated that a plurality of such information captureapparatuses 100 may be coordinated to monitor a larger space than aspace that can be covered by a single such apparatus. Thus, theapparatuses can be made aware of the presence of the other apparatuses,and may operate by transmitting all information to one of theapparatuses 100, or these apparatuses may each independently communicatewith remote data and computing location, which is adapted to piecetogether the various information received from the plurality of devices100, whether such information is prompted by information provided byinformation capture apparatus 100, or information is captured passively.These multiple apparatuses may be employed in a system allowing a userto log into any such system, or one in which tracking of a user throughthe fields of view of multiple devices may be desirable. Finally, it maybe possible for data to be transmitted along devices (i.e. daisy chain)to allow for transmission of data from a device that does not haveexcellent communication system service to one that does. Informationcapture apparatus 100 may also perform local processing on collectedinformation at information capture apparatus 100, and therefore forwardpre-processed information to remote data and computing location 300.Remote data and computing location 300 may also comprise a data storagerepository, or may be omitted, so that all processing is performed oninformation capture apparatus 100.

Referring next to FIG. 2, a more detailed view of an exemplaryembodiment of remote information capture apparatus 1000 (as an exampleof apparatus 100) and remote data and computing location 3000 (as anexample of location 300) is shown. As is noted in FIG. 2, apparatus 1000comprises an information capture device 1110 for capturing video andaudio data as desired. A motion detector 1115 or other appropriatetrigger device may be provided with capture device 1110 to allow for theinitiation and completion of data capture. Information capture device1110 may further comprise a visual or audio/visual data capture device,such as an audio/visual camera, or may be provided with an infrared,night vision, ultrasonic, laser, 2D, 3D, distance camera, radar or otherappropriate information capture device. Motion sensor 1115 may also beused as an information sensor, the collected motion sensing informationbeing provided with other collected information. Motion sensor may alsobe substituted with other sensors, including GPS sensors,accelerometers, rotational sensors, or other sensor related to thepatient employing apparatus 1000. A storage location 1120 is furtherprovided for storing captured information, and a processor 1130 isprovided to control such capture and storage, process collectedinformation, as well as other functions associated with the operation ofremote information capture apparatus 1000. An analysis module 1135 isprovided in accordance with processor 1130 to perform a portion ofanalysis of captured information at the remote information captureapparatus 1000. Apparatus 1000 is preferably further provided with adisplay 1140 for displaying information, and a data transmission andreceipt system 1150 and 1160 for communicating with remote data andcomputing location 3000.

Remote data and computing location 3000 preferably comprises systemmanagement functions 3030, and a transmission and reception system 3050and 3060 for communicating with apparatus 1000. Transmission andreception system 3050 and 3060 may further comprise various GPS modulesso that a location (if provided as a mobile device) of the device can bedetermined at any time, and may further allow for a message to be sentto one or more individual apparatuses 1000, broadcast to all apparatusesin a particular situation, or being used for administration of aparticular prescription regimen, of broadcast to all availableapparatuses. Remote computing and data location 3000 may be furtherprovided with data storage elements 3070 and processing elements 3080.Data storage elements 3070 preferably comprise one or more conventionalstorage units, and may be set up as a cloud computing system, or offlinestorage. Data storage elements 3070 are designed to receive theinformation collected above, and further to provide inputs of data intoprocessing elements 3080. Such elements may comprise individual centralprocessing units, graphical processing units, or other processingelements known to one of ordinary skill in the art. Remote computing anddata location may further include at least a processor and analysismodule, and a display, if appropriate. In accordance with an exemplaryembodiment, apparatus 1000 is adapted to be part of a system thatautomatically monitors progression of symptoms of a patient in a numberof ways, and may be employed during use of a medication adherencemonitoring system relying on visual, audio, and other real time orrecorded data. The system may similarly be employed to collectinformation from a user separate from use during medicationadministration. Users of apparatus 1000 (patients) in accordance withthe disclosure are monitored in accordance with their interaction withthe system, and in particular during medication administration orperformance of some other common, consistent activity, in response topresentation of visual material to the patient, or during the conduct ofan adaptive, automated interview with the patient in a manner asdescribed above with respect to FIGS. 16 and 17. Apparatus 1000 isadapted to receive instructions for patients from remote data andcomputing location 3000 and provide these instructions to patients. Suchinstructions may comprise written, video or audio instructions forguiding a patient to perform one or more activities, such as determiningwhether a patient is adhering to a prescribed medication protocol bypresenting a correct medication to the system, instructions and visualimages to be provided to the patient so that a response may be measured,or instructions that are adaptive in order to allow for the conduct ofan adaptive, automated interview with the patient.

Remote Information and Capture Apparatus

The described system therefore includes three main components, datacollection, data storage and analysis, and a mechanism for receivinguser input to pose a query, and to provide responses to the posed queryafter performing an appropriate data analysis. Referring next to FIG. 3,the data collection component will first be described. FIG. 3 depicts arelationship between an information hierarchy of information that isavailable from conventional sensor data associated with a standardremote information capture apparatus 100, including one or more sensorsfor the collection of conventional sensor data, and a hierarchy ofinformation that is available from advanced sensor data associated withan advanced remote information capture apparatus 100, including one ormore further advanced sensors for the collection of advanced sensordata, such as data that may be collected by remote information captureapparatus 100, is shown. FIG. 3 depicts sensors and data they capture,including sensors that capture behavioral data 305, self-reported data310, and physiological data 315, and additional advanced novelbiosensors and smartphone sensors that may monitor circuits or otherelectrical inputs 320, cells 325, molecules 330 or genes 335 and howthis collected sensor data may be integrated via connected sensors, onorder to fine one or more correlations between these sensors. By way ofexample, behavioral sensors that collect behavioral data 305 can be GPSsensors, accelerometers, or other sensors included within apparatus 100,Bluetooth connectivity and call logs, for example. Self-reportedinformation may be collected via real time surveys presented to apatient. A final conventional group of sensors for collectingphysiological data may include a heat rate monitor, a skin conductancemeter, a respiratory rate measurement device, or a mechanism forconfirming a startle reflex, by way of example. Additional novel bio andsmartphone sensors may use, e.g., real time EEG sensors (sensors 320),one or more mechanisms for performing direct testing on the cells 325 ofa patient, one or more molecular sensors (330), or one or more genechips or other gene sensing technology 335.

The depicted information relationship shows the sharing of informationand analysis in accordance with the data collected from standard sensorsources, such as GPS data, accelerometer data, self-reports and the likeand the more advanced sensors. While these big, commoditized andaccessible data sets are prevalent in use, they do not allow for indepth analysis that would support medical-grade solutions, such as theability to accurately monitor changes in symptoms of disease and todiagnose the existence and progression of disease. With the systems andtechniques disclosed herein, various symptoms, which are potentiallyindicative of a disease state, may be monitored, through the processingof known data sets. As shown, each potential symptom is tied to aparticular measurement. While these measurements may give some insight,there is no mechanism for combining these measurements, or for makingmore measurements based upon inventive tests, such as those proposed inaccordance with the present disclosure.

Therefore, in accordance with the various embodiments of the presentdisclosure, the systems and methods disclosed herein may provide fornovel testing and collection of advanced visual and other sensor data,and also for advanced analysis of the collected unique data. As will bedescribed below, collection of such data may be performed in both activeand passive modes, and may also allow unique, automated interaction witha user so that the data may be extracted in a most efficient manner,while tailoring the data collection process based upon the results ofearlier data captured during the process.

Once collected, data across multiple patients or users may be compiledto allow for further analysis. The collected information may be analyzedalong a plurality of dimensions so that expected changes along each ofthose dimensions may be determined. One or more patients may becharacterized as reacting to changes along one or more of thesedimensions. Other aspects of the demographic information for each of thepatients may provide the ability to recognize characteristic changesalong each of those dimensions categorized by the demographic of thepatient. Thus, by collecting information (e.g., patient symptominformation) across all patients, categorizing the data by demographic,and analyzing the categorized data, predictions of future expectedresponses across these same dimensions can be determined.

In a still further aspect of the present disclosure, potential futureexpectations of medication responsiveness may be determined based onpatient demographics by categorizing patient responses to theadministration of medication along the different demographic groupdimensions and by determining aspects of medication responsible forchanges to the characteristics along these same dimensions. The systemsand techniques disclosed herein can be used as a medicationidentification tool by searching a desired combination of patientresults to provide a guide for medications to be used to combat certainof the symptoms matching the dimensions.

In accordance with one or more embodiments, data is collected and may becorrelated to disease progression and allow for an in-depth analysis ofdisease as it progresses through to all stages and exhibits all symptomsof those stages. If such data is collected via an automated activeprocess, where one or more questions or other stimuli, are presented toa patient, a data sensitive process is employed where branching logicmay dictate not only a next question to be asked of a patient after aresponse to a prior question, but also may dictate the actual visual andother sensor data to be collected based upon a response to a priorquestion, data received from a sensor, or near real time, orasynchronous analysis of previously collected visual and other sensordata. In such a manner, in accordance with an exemplary embodiment, if apatient is monitored in an automated fashion to determine a response toa particular stimulus, such as being shown a particular image to elicita response, the system may output to a display and based upon a responseto the first image, a second image, in which the particular second imagechosen to be displayed depends on the specific response of the patientto the first image. Alternatively or in addition, if a user ingesting amedication pill is being monitored, and the system determines that theuser is having trouble swallowing the medication pill (e.g., where thesystem makes such a determination in an automated fashion in accordancewith artificial intelligence and computer vision analysis of datacollected by one or more data collection devices as described above),the system may focus the collection of high resolution video data ontoan area of the throat of the user in order to analyze micro-movements ofthe throat of the user to confirm actual ingestion.

In accordance with one or more exemplary embodiments, analysis and carepathways may be defined, and implemented in near-real time, while thesystem is automatically interacting with a patient during a singlesession (i.e. while the user is still interacting with the system, suchas in near real time, as opposed to a system that collects data during asession, analyzes the data offline, and then provides a response at alater time), for example, in order to efficiently guide the patient toperform a desired sequence of steps, and to focus data collection andthe steps to be performed to allow collection of data that may be mostrelevant to supporting analysis of a particular symptom or component ofdisease. For instance, a patient with Parkinson's disease, may warrantanalysis of changes in consistency of a hand tremor, for example, asidentified in an automated process in accordance with one or moreembodiments. Upon indication of a desire to monitor such a disease by aprovider of the monitoring system, which may include doctors, otherhealthcare providers, clinical trial sponsors, contract researchorganizations, and the like, data collection may be focused by thesystem on those aspects that are related to such hand tremor, in that ithas been predetermined that tremor is a correlated indicator toprogression of Parkinson's disease. In addition, for example, uponautomated identification of changes in or an absolute level of such handtremor based upon data collected by the system, different analysispathways may be initiated so that slight tremors or slight changes intremors, the system may output requests for the patient to performcertain sequences of actions, while testing of more intense tremors mayinvolve the system outputting requests for the patient to performdifferent sequences of actions to allow for different testing.Additional details of particular data to be collected will be describedbelow.

Aspects of the disclosure therefore allow for movement from a systemwhere known measurements are used to provide known insight to one inwhich novel measurements may be collected from a user, and novel,insight be gleaned from those novel measurements. The system describedin accordance with this disclosure devises a novel measurement, andbased upon this novel measurement, provides a novel insight. Thus, atleast the combination of novel measurements and insights differentiatesthe subject matter of the present disclosure from alternative systemsfor analyzing patient states. The systems and techniques disclosedherein may rely on both active and passive data collection via apparatus1000, for example, as will be further described below. Active datacollection may comprise, e.g., an automated interview (e.g., wherequestions are presented to a user and answers to the questions arerecorded), an interactive test, a self-assessment, or other interactionwhere material, stimulus, or prompt is actively pushed to the user toelicit a response to the material, stimulus or prompt. Passive datacollection comprises, e.g., automated video analysis while a user isperforming some other action, such as dosing, or administering theirmedication while engaging with a medication monitoring system (such as avisually based medication adherence monitoring system, such as thatprovided by AiCure®), using one or more sensors to collect data, orcollecting visual or other data at any time while the user is performingsome action not elicited solely for the purposes of collecting the testresponse data.

In addition to using the collected test response data as noted herein,the collected data may also be used to aid in confirmation of medicationadministration. Patient response to medication administration inaccordance with any of the measured parameters noted above may bepredicted, and then explored for new patients. Combining expectedresponses of any number of symptoms may be employed in order to providemultiple redundant layers of medication administration confirmation. Inaccordance with an exemplary embodiment of the present disclosure,sensors such as those used in a mobile phone or other device, includingtwo dimensional and three dimensional cameras (e.g., for time of flightcalculations) may be employed. Any of the following signals may bemonitored to confirm one or more elements indicative of propermedication administration: hand gestures, head tilt, Gulp (swallowing),movement of jaw, grip of pills, timing of performance of one or more ifthese actions, movement of shoulders, strain of the patient's neck,changes in breathing, pupil control, changes in blinking speed andconsistency, fluttering of eyes, any other indication of physicalstrain, and the like. Further in accordance with active analysis asnoted above, it may be further possible to monitor the following: 1)expressivity (including the level or amount of animation when smiling,frowning, etc.), 2) movement (tremors and motor control), 3)concentration (including gaze), 4) facial control (including eyemovement and blinks/twitches/paralysis), 5) cognition tasks (ability toread a paragraph), 6) visual breathing rates, 7) body mass changes, 8)facial wasting, and the like. These and other physiological changes mayalso be evaluated based on, for example, triangulation of skin using adepth sensor (a time of flight camera) to see flow of blood to measureclots or even heart rate. Blood pressure may be measured by scanning howthe face of the patient has changed, or by looking beneath the skin tomonitor capillary flow or blockage. Changes in such abilities may betherefore be indicative or progression of disease, and effectiveness ofmedication administration.

Additionally, in accordance with the second aspect of the invention, aswill be described below, the collected data from any of these sourcesmay be captured for more advanced analysis, including analysis basedupon predetermined requests for information included in remote data andcomputing location 3000, or in response to near-real time requests froma user via remote information and display device 400.

Referring next to FIGS. 4 and 5, details of active mechanisms forcollecting data from a patient will be described. These mechanismsshould be considered examples, noting that any reference to numbers ofpresentations, or any particular language employed is exemplary only,and that other questions, images or the like may be employed. As isshown in FIGS. 4 and 5, at step 510, a user is first preferablyrequested to position their face in the middle of a field of view of animage captured by a camera and as displayed on a display of a device,such as device 1000 (see FIG. 1). The purpose for such placement inaccordance with an embodiment of the invention may be to determine abaseline visual view of the face of the user. As will be described belowin greater detail, this baseline may be determined from a singleimage/video collection or may preferably be determined in accordancewith a plurality of image/video baseline collections, and in a manner asdescribed herein. If determined in accordance with such a plurality ofimage/video baseline collections, if is also therefore possible todetermine whether the current baseline is in line with the othercollected baselines, or whether the user is already showing differencesfrom their average baseline, and a transformation or adjustment shouldbe made.

Once the baseline has been collected, as shown in FIGS. 4 and 5, a firstquestion from a binary branch of questioning logic is asked at 520. Asequence of predetermined questions or a branching logic associated withthe baseline questions may be presented as an instrument designed todiagnose a particular disease or determine progression of disease or oneor more symptoms associated therewith. The sequence of questions isimportant, as determined by the inventors of the present invention, inorder to provide a more natural interview process to the user. Thus, insome implementations, a relatively broad question is asked first. Theuser is then asked to answer the question using the touch screenbuttons, and verbally. In an exemplary embodiment, video may also berecorded of the user to allow for subsequent analysis of that video toassist in determination of progression of symptoms, or diagnosis ofdisease.

At step 530, based upon the response provided at step 520, an additionalquestion may be asked in an automated fashion, requesting additionalinformation in response to the prior question. In the example shown inFIGS. 4 and 5, if the user answers yes to whether there were anyactivities that the user enjoyed over the weekend, the user may then beasked to respond, using the touch screen of device 510, a questiondesigned to broaden the ideas that they are thinking about and may alsobe prompted to provide a verbal or visual response. For example, theuser may be asked to select, from a list of activities, an activity thatthe user enjoyed. This question allows for active monitoring of otheractions or characteristics of the user after branching logic is applied.After completion of the broadening question, a more focused question isnoted at step 540 in order to draw the user further into theconversation. The response at this point is preferably verbal andvisual. Finally, after providing the focused response, the user may beasked to elaborate in order to further draw the user into a conversationand avoid the impression of a system that is simply recording.

The sequence of questions may be based upon any prior questions and mayalso be based upon analysis of the audio and video responses receivedfrom the current patient using the system. While the answers to thequestions may provide some information about the user, in accordancewith an exemplary embodiment of the present disclosure, it is theanalysis of the audio and video responses that is most important.Indeed, the actual wording provided by the user may be unimportant.Rather, by drawing the user in to a conversation in an automatedfashion, the system is able to further analyze the received audio andvideo sequences in order to determine mood, feelings, progression ofsymptoms of any sort, or development of disease based upon thesesymptoms. How the user performed any set of tasks such as interactingwith the touch screen or other requested tasks is as important as thetypes of answers provided. Additionally, the method in which theresponses were described can provide significant insight into thecurrent mood of the subject.

The systems in accordance with an embodiment of the present disclosureare therefore enabled to conduct automated, intelligent interviews, andto also extract and analyze visual, temporal, gestural, prosodic &natural language data, as will be described below. As also noted above,intelligent interviews may be performed by asking questions of the user,evaluating responses, and branching logic based upon the responses. Insuch a manner, a realistic interview is provided in which the user isencouraged to engage most completely with the inventive system to allowfor the greatest amount of analyzable data to be collected.

In accordance with an alternative embodiment, rather than having theuser interact directly with the device 1000, an interviewer may relyupon device 1000 to present to the interviewer the questions to be askedof the user. The responses to these questions may be provided verbally,or also including video capture. Thus, after the interviewer asks aquestion, the audio and/or video responses may be collected and analyzedin near-real time, thus allowing for a next question to be presented tothe interviewer to ask the user. The analysis of the audio and video maysimilarly be performed at a remote location (3000), thus allowing for asecond, more in-depth analysis, or may be provided in real time ifconnectivity is good, and the local device with which theinterviewer/user is interacting is unable to properly process theinformation.

Remote Data and Computing Location

The data noted above and as further described below is preferablycollected to create a database that may be indexed by any number ofdimensions, but preferably and initially by disease indication. While itis possible that all processing may take place at a local device of theuser, or alternatively all processing may take place at a remotelocation with the mobile or other local device only acting as a datacapture device, interaction with a local device of the user allows fornear-real time interaction with the user, even if there is nocommunication network connection, and further in depth analysis at alater time in an asynchronous manner. Subsequent indexing may beperformed by patient demographics and patient response to medicationadministration. When a new patient is introduced to the platform to bemonitored taking their medication, the system can first be employed toidentify any particular risk that the patient may encounter, based uponany indexed dimension of the patient. The system therefore provides anearly warning system related to potential patient response to aparticular medication. Therefore, not only can the system be used tomonitor an upcoming patient into the system taking particularmedication, but also can be used to determine a potential response of atheoretical patient to a particular medication to be administered. Inthis manner, quality of response (whether positive, indicative ofefficacy of a medication, or negative including the likelihood of anadverse reaction) to a particular medication may be determined.

Other aspects of the patient response to the medication may also bepredicted based upon the patient demographics and the database of priorpatients' responses, and the prior response of the particular nextpatient, if that patient is already in the database. Thus, diseasesymptom progression may be predicted, and may be adjusted based uponexpected medication response. Deviation from the expected symptomprogression as determined in an automated fashion by the present systemsmay indicate a parallel deviation from the required medicationadministration protocol, or an atypical response of the patient to themedication. Further, a predicted impact (i.e. comparing a measured valueor values to an expected value or values as determined from prior datacaptures) on the computational diagnostic measures described above,whether collected actively or passively, may be provided. The results ofsuch an analysis may be converted to an efficacy score, indicative ofthe effectiveness of the medication based upon the various dimensions ofthe patient. Values may be combined across measurements into a singlescore indicative of effectiveness of the medication. Two drugs, one thatgreatly improves tremor, but does poorly with fine motor control, andanother that is the opposite may have similar efficacy scores, becausethey both improve conditions. The true benefit of the system is to tryto recognize the features of the first drug that improve tremor and thefeatures of the second drug that improve fine motor control, and searchfor a new drug having these two features so that a better drug withhigher efficacy score overall can be discovered. Thus, for futurepatients, measurement of a number of parameters may allow for theprediction of how effective a medication may be for a particularpatient, and ultimately may allow for the selection of one medicationover another based upon the demographics or other measurements of thesubject, and similarly may be able to predict the response of thepatient to the measurements noted above. Such comparisons may beperformed during the entire term of medication administration, and mayallow for monitoring of disease progression, and perhaps suggest whenchanges in medication protocol may be justified. Once accumulated, sucha database may be available for use in order to aid in predictingpatient response to other medications that may be under development. Forexample, a scoring system may be created to show effectiveness andsafety of a particular medication in comparison to predecessor drugs orother treatment options that have gone through the system when presentedwith a particular disease or therapeutic area. Additionally, a measuredresponse of patients to a particular drug may allow for prediction as tohow similarly-situated patients may respond to another drug, oralternatively, how other patients may respond to the same drug. Throughanalysis, other patients having similar responses on one or moreparticular dimensions may allow for prediction of successful response tothe drug, for example. In this manner, predictability of response to adrug may be available based upon similarity of patients on the mostcritical dimensions.

In accordance with alternative embodiments, rather than simplydetermining one or more correlations between patient demographics andexpected responses to the one or more testing scenarios noted above,similar correlations may be determined to one or more gene expressionprofiles. Gene expression profiling is a high throughput approach toanalyze the expression of tens of thousands of genes simultaneously.Expression of specific groups of genes, or gene expression profiles, canthen be correlated to pathologic diagnosis, clinical outcomes, ortherapeutic response. Transcriptional profiling experiments can be usedto generate compendia of gene expression data across different celltypes, development times, and in response to distinct stimuli. A similarapplication of this determined gene expression and gene transcriptionalprofile in accordance with one or more embodiment of the presentdisclosure correlates to movement signatures, symptom progression, orother expected or observed responses in accordance with the active orpassive measurements noted above, based upon one or more individual orsequentially-applied measurements.

In accordance with another aspect, through the use of remote informationand display device 400, not only can collected data be pre-processed toprovide one or more predetermined analysis results, the system maypreferably be adapted to receive user input data from an external userdirecting one or more queries related to the collected data. Thus, inaccordance with one or more embodiments, a user may present a patientprofile to the system via a remote information and display device 400and evaluate how the patient may respond to a plurality of medications.As noted above, a plurality of medications and the response of aplurality of patients may have been previously captured. The profile ofeach patient may be dissected into a plurality of components related totheir improvement over a plurality of different dimensions, and alsobased upon simpler demographics, or more complex disease state orcharacteristics, and each of these components catalogued. Thus, when auser queries the system, it is possible to provide a demographic profileof a potential user and determine how that user might react or respondto any number of these stored medications.

Furthermore, in addition to dissecting the profile of each patient, theprofile of each medication may similarly be dissected. Thus, eachmedication may be measured along a plurality of dimensions in order todetermine characteristic of that medication. These measuredcharacteristics may then be used to predict which medications may beapplicable to particular disease or patient populations. Thus, by way ofexample, measured medication characteristics may include 1) theirsedative effect, 2) the ability to reduce paranoia, and/or 3) ease ofapplication, among others. If one is then looking to find a medicationfor use in a particular population, medications that have a low sedativeeffect while reducing paranoia, and that are easy to administer, mayprove to be a winning combination. Drugs having these characteristicscan then be investigated as potentially effective in this population.

In this manner, users may be given access to remote information anddisplay device 400 as an analysis tool that interacts with and performsrequested analysis on the data stored in remote data and computinglocation 300 after collection by remote information and capture device100. This interface into the complex system preferably provides a simpleyet robust system to determining potential patient response, and fordrug discovery, without the need for the user to collect, understand, orindependently analyze the collected data. Rather, a simple interfaceallows these users to easily pose complex queries to the data and bepresented with results determined through the user of a complexartificial intelligence-based system.

The following description of analysis of users may be performed inaccordance with the active presentation of information (as described)but may also be utilized in accordance with a passive data collection.In such a passive environment, identification of the timing of triggersis more important, as they are not predetermined. The describedcollection of data is an example of collection of data to be provided toremote data and computing location 300.

Data Collection System Example

Referring next to FIG. 6, a system for designating action units on theface of a user is shown. In FIG. 6, multiple different points (e.g., 68different points) on the face of a user are defined. While anyparticular points may be defined, the number and location of the pointsmay be selected based on the desired attributes of the user to analyze.For instance, the 68 points shown in FIG. 6 (also known as keypoints orlandmarks) allow for robust tracking of action units of the face, andthus providing the ability to analyze facial expression withoutoverwhelming the system. Each of these points may be measured todetermine its movement during performance of one or more activities,such as in response to the presentation of one or more stimuli to theuser. As is known in the art, when a particular action unit moves(action units may comprise a single or multiple points, keypoints orlandmarks), it is possible to measure this movement. However, if theaction unit moves too far from an expected “neutral” location, thesystem breaks down and the action unit cannot be recognized. However,rather than using as a “neutral” location system an average mask acrossall users, the system of the present disclosure relies on a morecustomized mask for the individual user. Thus, by setting a baselinepositioning of the action units of a particular user, it is possible tobetter account for differences between the face of a particularindividual. Such a baseline position may be defined by providing a basicset of images or other material to a user and measuring the response ofthe predefined action units. By presenting this consistent “calibration”set of images or other stimuli it is possible to then determine anexpected movement of the action units, and then determine a relativemovement when further unique or different images or stimuli areprovided. This allows for a user with relatively low action unitmovement to be judged against this expectation, while an animated personwill be judged against this more animated expectation. Thus, theexpected movements may also be tied to a particular individual, thusallowing for more flexibility in tracking the action units as thesubject is provided with one or more stimuli.

Also, it has been discovered that it is possible to identify which ofthe presented action units (or other appropriate measurement points orvalues, including but not limited to keypoints, landmarks (asrepresented by the points in FIG. 6, and preferably comprising one ormore points on the face of a user that may be indicative of movement ofthe face that may be of interest to a reviewer, typically indicative ofchanges in facial expression), shapes, textures, poses, features fromboth 2D and 3D sensors) are most likely to be important when reviewingprogressions for a particular disease or symptom, for example. Basedupon the desired action units, it is therefore to focus on only theseaction units, and not measure the others. Thus, a context-sensitivesystem may be provided in which a priori knowledge about a particulartherapeutic area or disease state may allow for the focusing of thesystem on the action units most likely to give valuable information.Additionally, it is also possible, in accordance some embodiments, tovary the stimuli presented to the user in accordance with the desiredinformation to be extracted, and also based upon therapeutic area,disease state, symptom progression, or the like. By way of example, itis possible to present images to individuals suffering fromschizophrenia who typically present what is referred to as negativesymptoms. They exhibit very low levels of action unit movement, andgenerally not an overly animated response. Because generating a responseis the goal, extremely happy or disturbing images may be shown to apatient. For others with more standard response expectations, moremainstream images may be more appropriate. This feature selection stepcan be achieved through a combination of expert identifications andmachine learning based approaches to derive more effective butnon-intuitive features. This process may also be applied to othermeasurable quantities, such as level of tremor, voice inflection,volume, etc.

In accordance with some embodiments, daily or more frequent images of auser responding to baseline images may be taken for a baselinedetermination (step 510 above) in order to evolve the current baselinefor a particular individual. The response of the patient to thesebaseline images may be averaged to obtain a customized baseline againstwhich other collected patient responses to the presentation of future(e.g., the same images as the baseline images or different images fromthe baseline) images and action unit movement can be compared to measurechanges in such movements. Furthermore, in addition to providing acustom baseline for each individual user, it is also possible to comparethis customized baseline to an average baseline across all subjects inorder to discern absolute information about the current status of theuser relative to others in the same therapeutic area, or suffering forthe same disease, by way of example. Therefore, daily monitoring of theuser performing a repetitive task, such as administering a medicationpill (for example) provides a visual baseline. Through observation of auser over time performing such a standard repetitive task, a baselinemay be provided against which future data collection can be compared.Repetitive tasks also help identify the core symptom-related featuresand remove noise and variations in natural behaviors. The baseline maytherefore not be based on just one measured instance, but may insteadcomprise a model learned from multiple instances of measurement. In thismanner, deltas from an individualized baseline, rather than an averagebaseline can be determined.

When a user interacts with the system of the present disclosure, thesystem may present the user with one or more types of stimuli (e.g.,audio stimuli and/or visual stimuli). Such stimuli may comprise, e.g., aquestion or other prompt, as noted above, or may comprise thepresentation of some visual or audio material that elicits a responsefrom the user. Referring next to FIG. 7, a graph 700 representative of aresponse of a user to the presentation of just such a stimulus is shown.In the sequence of events leading up to graph 700 of FIG. 7, the systemoutputs to a display an image 710 for viewing by the user, in this casean image of a car crash. Traces 740 depict an amount of movement of oneor more of the action units (1-68) shown in FIG. 6 from a baselineposition. Thus, for approximately 1400 frames of video (x-axis) themovement of these action units is traced along the y-axis in the presentexample. A number of the 1400 frames are recorded prior to thepresentation of image 710, which takes place at line 720. As is shown inFIG. 7, sometime after the presentation of the image 710 at line 720,the movement of the action units of the user are reduced, on average, ascompared with the time prior to line 720. The system is therefore ableto track all 68 points (or keypoints or landmarks) on the face of theuser, deriving 17 facial action units (points that work together and areindicative of movement of a muscle structure of the face, such as whenparticular action units are engaged when a subject smiles, frowns,cries, etc.), determining 18 gesture classes (gestures that can beconfirmed based upon actuation of one or more particular action units),and also employing gaze tracking to trace the direction of gaze of theuser. By grouping the collected data on the movement of the data points,the action units can be determined. Movements of the action units can beanalyzed to determine which are associated with the performance of oneor more gestures, and those gestures can then be determined based uponmonitoring movement of the points on the face of the user.

Moreover, it has been determined by the inventors of the presentinvention that the time between line 720 and a next line 730(approximately 1000 milliseconds) is a portion of the timeline where theuser's mind and body may present a response to the stimulus, but wherethe user has not yet consciously recognized the stimulus, or the user'sresponse to this stimulus. Beyond this time, and thoughtful action takesover and guides the response of the user. It is this portion of thetimeline (and more precisely, the first 250 milliseconds afterpresentation of the stimulus) that may provide the greatest insight intouser response. This unique measurement is not able to be performedmanually, as the human vision system is not able to see, recognize andremember facial movements over such a short timeline. Therefore, in someimplementations, it may be advantageous to view the facial movements ofthe user using high speed photography at a frame rate more than doublethe common 24 fps frame rate, and preferably higher to allow for thecollection of extremely short movements or gestures. Such a camera maybe employed as part of a typical mobile device, or may be included as anadd on, via USB or other common connector to the mobile device. Thesystem may further be provided as a standalone system that includes asingle purpose high speed camera, connected to a computer capture devicein a traditional manner. in accordance with the embodiments of thepresent disclosure so that these small timeframes may be analyzed. Thesemovements of the face in this minimal time frame are referred to as“microexpressions.” Essentially, these microexpressions are presented bythe face of the user before the user is fully cognitive of and able toprocess the received stimulus. By varying the window over which onelooks at these microexpressions, the system can filter for differentresponses.

By accumulating multiple responses of a user to multiple stimuli, it ispossible to build a profile of the user that allows for more completeunderstanding of the user's status related to symptoms and disease. Itis important, as noted above, and as is further shown in FIG. 8A,determining a proper baseline for each user of the system allows formore customized review of future actions by that user. As is shown inFIG. 8A, a first baseline reading during a face identification may beperformed at 810, generating a baseline analysis time sequence 820across a timeline of any number of days, or other time measurement. Thisbaseline analysis may also be performed while the user is speaking orperforming an action, such as any action that may be considered a“background” action or situation (i.e. portions of the image that arenot determined to be part of the user action being recorded, such as thewall, moving cars, etc. Thus, to remove the effects on measurements fromsimple talking, the user can be asked to perform a standard action for abaseline while talking. To remove the effects of walking, the user canbe asked to set a baseline while walking, etc. As is shown in timesequence 820, initial baseline is consistent over much of the graph, butchanges at point 830, indicating a systematic change for this particularuser. Thus, a change in baseline of the user is quite evident, and maybe investigated. Graph 840 depicts the material presented for each timeperiod in the graph 820, and includes readings for a number of actionunits being measured. FIG. 8B depicts a resulting image afternormalizing the baselines for all of the action units that seemed tocorrespond to a marked change at frame 200 (x-axis), we can begin to seea distinct change from baseline after the image is shown and before theuser begins talking, as previously discussed with respect to FIG. 7. Theline at frame 200 (x-axis) represents is post-reveal assuming 30 fpsvideo rate.

Referring next to FIG. 9, an analysis hierarchy is shown, and includingbuilding upon the collected information to analyze input and collecteddata. As is shown, a baseline shift analysis is first performed, asdescribed above, at level 910. Next, at level 920, a video contentanalysis is performed (such as that shown above with respect to FIG. 7,indicating changes in the face of the user. Additionally, at level 930,a prosodic analysis of spoken video of the user may be performed. Theseinputs may be combined to note gross movement 940, illustrators 945 andmanipulators 950. These levels of information may in turn be used todetermine affect, for example, such as microexpressions 955, expressions960 and attitudes 965, resulting in a sentiment analysis. Such asentiment analysis ultimately allows for the analysis and determinationof the sentiment currently held by a user, and may do so in an automatedfashion, relying on the automated processes noted above. Finally, atemporal analysis 970 may be performed to determine how the sentiment ofthe user may change over time.

FIG. 10 depicts graphically the analysis proposed in FIG. 9. As is shownin FIG. 10, longitudinal analysis (looking at data over time, and overmultiple instances of data collection across multiple data types) ofcompound data sources may be used to analyze visual and speech data overtime to determine sentiment. As is shown in FIG. 10, an image revealpoint corresponding to the point in time at which the system exposes auser to a stimulus, such as an audio or visual stimulus, may allow forthe determination of microexpression, as noted above. As time passes,different expressions may be determined, after the period formicroexpressions has passed, as also noted above. These microexpressionsand expressions may be analyzed to determine attitudes of the user, andultimately sentiment. By first determining action unit movement, andthen facial expression, one is able to further determine attitudes ofthe users in response to a stimulus (i.e. are they happy when they see apicture of a cute dog), and finally figure out what sentiment they mayhold around the subject. These analyses are performed by collectingsignificant amount of data from subjects with determined baselines, anddetermining which action unit movements are ultimately indicative of thethoughts, feelings and sentiments of the users. After an additionalperiod of time, it may be beneficial to switch to vocal prosodicanalysis in order to determine additional expressions.

Referring next to FIG. 11, a graph such as that shown in FIG. 7 isprovided, but is limited to the use of action units 6 and 12, as thesewere determined to be relevant for analysis in this situation. As can beseen, between the vertical lines at 200 and 250 frames lies themicroexpression response. At approximately frame 300, speech begins, andthe expression (in this case happiness) is greatly repressed. FIG. 12depicts the same measurement for a greater period of time. Of note arethe spikes denoted with action that may have caused them. It is clearthat these are not expressions of the user. Therefore, in accordancewith some embodiments, and as shown in FIG. 13, it is desirable tofilter the waveform to remove such spikes to provide a more consistentwaveform related to the actual action unit movement. FIG. 14 depictsthat use of multiple action units to determine an action may result insignificant noise. FIG. 14 depicts a comparison of the noise between theuse of 7 action units and two action units. It is therefore desirable touse fewer action units to make a determination when possible.

Finally, FIG. 15 depicts the use of prosodic analysis, but combined withthe inventive visual analysis. By knowing what the user is looking at,it is possible to provide additional context, thus improving theaccuracy of the prosodic readings. For example, when a user is lookingat something providing the user happiness, baseline for the prosodicanalysis may be altered to give a more accurate response. Prosodicanalysis involves analyzing the speech patterns of a person. Bydetermining the speech patterns of an individual, it is possible tomeasure changes over time of that individual, and to make rudimentarydeterminations about progression of symptoms or disease. With thepresently disclosed systems and techniques, it is possible to correlateprosodic analysis with video analysis of the face of a patient. In someimplementations, correlation of movement of action units and prosodicanalysis in response to presentation of images or other stimuli to anindividual allows for any correlations or lack thereof to be determined.Differences in these correlations between individuals may providefurther details of the individual's response and progression of disease.These additional data points allow for a far more complex and extensiveanalysis, in the manner as described in FIG. 15, which depictsmeasurement of pitch, velocity and acceleration of the speech pattern ofan individual. As is shown, countdown beeps are provided, and aftercomplete, a narrator asking a question, for example, is provided to theindividual. The graph then depicts the individual response stating“army” and “serious.” The analysis of this responsive speech patternprovides a benchmark for comparison to other individuals, and a baselinefor comparison to future analysis of the same user. By the use ofprosodic in conjunction with visual analysis data, more specificresponse patterns and progression of disease and symptoms may bedetermined.

Therefore, in accordance with the various embodiments of the invention,improved methods and systems are provided for conducting interviews witha user, for analyzing collected video and audio data, and fordetermining sentiment of the user employing microexpressions and otheranalysis techniques.

All or part of the processes described herein and their variousmodifications (hereinafter referred to as “the processes”) can beimplemented, at least in part, via a computer program product, i.e., acomputer program tangibly embodied in one or more tangible, physicalhardware storage devices that are computer and/or machine-readablestorage devices for execution by, or to control the operation of, dataprocessing apparatus, e.g., a programmable processor, a computer, ormultiple computers. A computer program can be written in any form ofprogramming language, including compiled or interpreted languages, andit can be deployed in any form, including as a stand-alone program or asa module, component, subroutine, or other unit suitable for use in acomputing environment. A computer program can be deployed to be executedon one computer or on multiple computers at one site or distributedacross multiple sites and interconnected by a network.

Actions associated with implementing the processes can be performed byone or more programmable processors executing one or more computerprograms to perform the functions of the calibration process. All orpart of the processes can be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) and/or an ASIC(application-specific integrated circuit). Other embedded systems may beemployed, such as NVidia® Jetson series or the like.

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only storagearea or a random access storage area or both. Elements of a computer(including a server) include one or more processors for executinginstructions and one or more storage area devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from, or transfer data to, or both,one or more machine-readable storage media, such as mass storage devicesfor storing data, e.g., magnetic, magneto-optical disks, or opticaldisks. Processors “configured” to perform one or more of the processes,algorithms, functions, and/or steps disclosed herein include one or moregeneral or special purpose processors as described herein as well as oneor more computer and/or machine-readable storage devices on whichcomputer programs for performing the processes are stored.

Tangible, physical hardware storage devices that are suitable forembodying computer program instructions and data include all forms ofnon-volatile storage, including by way of example, semiconductor storagearea devices, e.g., EPROM, EEPROM, and flash storage area devices;magnetic disks, e.g., internal hard disks or removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks and volatilecomputer memory, e.g., RAM such as static and dynamic RAM, as well aserasable memory, e.g., flash memory.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's device in response to requests received from the web browser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface or a web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data, e.g., an HTML page, to a userdevice, e.g., for purposes of displaying data to and receiving userinput from a user interacting with the user device, which acts as aclient. Data generated at the user device, e.g., a result of the userinteraction, can be received from the user device at the server.

In addition, the logic flows depicted in the figures do not require theparticular order shown, or sequential order, to achieve desirableresults. In addition, other actions may be provided, or actions may beeliminated, from the described flows, and other components may be addedto, or removed from, the described systems. Likewise, actions depictedin the figures may be performed by different entities or consolidated.Furthermore, various separate elements may be combined into one or moreindividual elements to perform the functions described herein.

While visual and audio signals are mainly described in this invention,other data collection techniques may be employed, such as thermal cuesor other wavelength analysis of the face or other portions of the bodyof the user. These alternative data collection techniques may, forexample, reveal underlying emotion/response of the patient, such aschanges in blood flow, etc. Additionally, visual depth signalmeasurements may allow for capture subtle facial surface movementcorrelated with the symptom that may be difficult to detect with typicalcolor images.

Other implementations not specifically described herein are also withinthe scope of the following claims. For example, the actions recited inthe claims can be performed in a different order and still achievedesirable results. As one example, the processes depicted in theaccompanying figures do not necessarily require the particular ordershown, or sequential order, to achieve desirable results. In some cases,multitasking and parallel processing may be advantageous.

It should be noted that any of the above-noted inventions may beprovided in combination or individually. Elements of differentembodiments described herein may be combined to form other embodimentsnot specifically set forth above. Elements may be left out of theprocesses, computer programs, etc. described herein without adverselyaffecting their operation. Furthermore, the system may be employed inmobile devices, computing devices, cloud based storage and processing.Camera images may be acquired by an associated camera, or an independentcamera situated at a remote location. Processing may be similarly beprovided locally on a mobile device, or a remotely at a cloud-basedlocation, or other remote location. Additionally, such processing andstorage locations may be situated at a similar location, or at remotelocations.

While operations are depicted in the drawings in a particular order,this should not be understood as requiring that such operations beperformed in the particular order shown or in sequential order, or thatall illustrated operations be performed, to achieve desirable results.In certain circumstances, multitasking and parallel processing may beadvantageous. Moreover, the separation of various system modules andcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

What is claimed is:
 1. A system for user-initiated analysis of changesin symptoms of a patient, comprising: a storage device for storing datacaptured of a patient performing an action in response to a stimuluspresented to the patient, the stimulus presented to the patientcomprising one or more instructions provided to the patient inaccordance with monitoring of medication administration of the patient,and wherein the action performed in response to the stimulus presentedto the patient comprises the action of the patient performing arequested step indicative of proper medication administration and isindicative of a change in one or more symptoms associated with a diseaseand the overall progression of the disease, the storage device furtherstoring one or more item selected from the group of: demographicinformation of the patient, disease progression information of thepatient, and one or more medication characteristics of the medicationsubject to the medication administration monitoring process; and thestorage device further storing demographic information of a plurality ofpatients, disease progression information of the plurality of patients,and medication characteristics of medication used by the plurality ofpatients; a user input module for receiving input from a user indicativeof a request for information; a processor for determining, in responseto the user input request for information, and in accordance with thecaptured data of the patient performing the action in response to thestimulus, a correlation between two or more of the following: one ormore of the determined stored reactions, the demographic information ofthe patient, the demographic information of the plurality of patients,the disease progression information of the patient, and the one or moremedication characteristics of the medication, the correlation being usedto determine efficacy of the medication; and an output module forreturning to the user information comprising the determined correlation.2. The system of claim 1, wherein the input from the user comprises anindication of a potential patient and a proposed medication.
 3. Thesystem of claim 2, wherein the potential patient is defined by at leastone or more items of demographic information and one or more diseasecharacteristics.
 4. The system of claim 3, wherein the determinedcorrelation comprises one or more predictions of a reaction of thepotential patient to the proposed medication.
 5. The system of claim 1,wherein the input from the user comprises an indication of a proposedpatient and a potential medication.
 6. The system of claim 5, whereinthe potential medication is defined by at least one or more medicationcharacteristics.
 7. The system of claim 6, wherein the processor furthercompares the one or more medication characteristics with medicationcharacteristics of medications stored to the storage device.
 8. Thesystem of claim 7, wherein the processor further determines one or moremedication characteristics of the potential medication that havepreviously reacted in a positive manor for prior patients havingdemographic information similar to that of the proposed patient.
 9. Thesystem of claim 8, wherein the processor further determines whether thepotential medication is likely to generate a positive outcome for theproposed patient.
 10. The system of claim 1, wherein the input from theuser comprises an indication of a proposed disease defined by at leastone or more disease characteristics.
 11. The system of claim 10, whereinthe processor compares the one or more disease characteristics to one ormore medication characteristics for one or more medications stored tothe storage device in order to identify one or more medicationcharacteristics that, when applied to a patient, are likely to result ina positive progression of the disease in the patient.
 12. The system ofclaim 11, wherein the processor identifies one or more medicationscomprising the one or more medication characteristics as likely, whenapplied to a patient, are to result in a positive progression of thedisease in the patient.
 13. A system for user-initiated analysis ofchanges in symptoms of a patient, comprising: a storage device forstoring data captured of a patient performing an action in response to astimulus presented to the patient, the action being indicative of achange in one or more symptoms associated with a disease, the storagedevice further storing one or more item selected from the group of:demographic information of the patient, disease progression informationof the patient, and one or more medication characteristics of amedication being administered to the patient; a user input module forreceiving input from a user indicative of a request for information; aprocessor for determining, in response to the user input request forinformation, and in accordance with the captured data of the patientperforming the action in response to the stimulus, a correlationbetween: one or more of the determined stored reactions, the demographicinformation of the patient, the disease progression information of thepatient, the one or more medication characteristics of the medicationbeing administered to the patient, and the efficacy of the medication inthe patient and an output module for returning to the user informationcomprising the determined correlation.
 14. The system of claim 13,wherein the input from the user comprises and indication of a potentialpatient and a proposed medication, wherein the potential patient isdefined by at least one or more items of demographic information and oneor more disease characteristics, and wherein the determined correlationcomprises one or more predictions of a reaction of the potential patientto the proposed medication.
 15. The system of claim 13, wherein theinput from the user comprises an indication of a proposed patient and apotential medication, wherein the potential medication is defined by atleast one or more medication characteristics, and wherein the processorfurther compares the one or more medication characteristics withmedication characteristics of medications stored to the storage device.16. The system of claim 15, wherein the processor further determines oneor more medication characteristics of the potential medication that havepreviously reacted in a positive manor for prior patients havingdemographic information similar to that of the proposed patient, andwherein the processor further determines whether the potentialmedication is likely to generate a positive outcome for the proposedpatient.
 17. The system of claim 13, wherein the input from the usercomprises an indication of a proposed disease defined by at least one ormore disease characteristics.
 18. The system of claim 17, wherein theprocessor compares the one or more disease characteristics to one ormore medication characteristics for one or more medications stored tothe storage device in order to identify one or more medicationcharacteristics that, when applied to a patient, are likely to result ina positive progression of the disease in the patient.
 19. The system ofclaim 18, wherein the processor identifies one or more medicationscomprising the one or more medication characteristics as likely, whenapplied to a patient, are to result in a positive progression of thedisease in the patient.